Follow-Me Project

Congratulations on reaching the final project of the Robotics Nanodegree!

Previously, you worked on the Semantic Segmentation lab where you built a deep learning network that locates a particular human target within an image. For this project, you will utilize what you implemented and learned from that lab and extend it to train a deep learning model that will allow a simulated quadcopter to follow around the person that it detects!

Most of the code below is similar to the lab with some minor modifications. You can start with your existing solution, and modify and improve upon it to train the best possible model for this task.

You can click on any of the following to quickly jump to that part of this notebook:

  1. Data Collection
  2. FCN Layers
  3. Build the Model
  4. Training
  5. Prediction
  6. Evaluation

Data Collection

We have provided you with a starting dataset for this project. Download instructions can be found in the README for this project's repo. Alternatively, you can collect additional data of your own to improve your model. Check out the "Collecting Data" section in the Project Lesson in the Classroom for more details!

In [6]:
import os
import glob
import sys
import tensorflow as tf

from scipy import misc
import numpy as np

from tensorflow.contrib.keras.python import keras
from tensorflow.contrib.keras.python.keras import layers, models

from tensorflow import image

from utils import scoring_utils
from utils.separable_conv2d import SeparableConv2DKeras, BilinearUpSampling2D
from utils import data_iterator
from utils import plotting_tools 
from utils import model_tools

FCN Layers

In the Classroom, we discussed the different layers that constitute a fully convolutional network (FCN). The following code will introduce you to the functions that you need to build your semantic segmentation model.

Separable Convolutions

The Encoder for your FCN will essentially require separable convolution layers, due to their advantages as explained in the classroom. The 1x1 convolution layer in the FCN, however, is a regular convolution. Implementations for both are provided below for your use. Each includes batch normalization with the ReLU activation function applied to the layers.

In [7]:
def separable_conv2d_batchnorm(input_layer, filters, strides=1):
    output_layer = SeparableConv2DKeras(filters=filters,kernel_size=3, strides=strides,
                             padding='same', activation='relu')(input_layer)
    
    output_layer = layers.BatchNormalization()(output_layer) 
    return output_layer

def conv2d_batchnorm(input_layer, filters, kernel_size=3, strides=1):
    output_layer = layers.Conv2D(filters=filters, kernel_size=kernel_size, strides=strides, 
                      padding='same', activation='relu')(input_layer)
    
    output_layer = layers.BatchNormalization()(output_layer) 
    return output_layer

Bilinear Upsampling

The following helper function implements the bilinear upsampling layer. Upsampling by a factor of 2 is generally recommended, but you can try out different factors as well. Upsampling is used in the decoder block of the FCN.

In [8]:
def bilinear_upsample(input_layer):
    output_layer = BilinearUpSampling2D((2,2))(input_layer)
    return output_layer

Build the Model

In the following cells, you will build an FCN to train a model to detect and locate the hero target within an image. The steps are:

  • Create an encoder_block
  • Create a decoder_block
  • Build the FCN consisting of encoder block(s), a 1x1 convolution, and decoder block(s). This step requires experimentation with different numbers of layers and filter sizes to build your model.

Encoder Block

Create an encoder block that includes a separable convolution layer using the separable_conv2d_batchnorm() function. The filters parameter defines the size or depth of the output layer. For example, 32 or 64.

In [9]:
def encoder_block(input_layer, filters, strides):
    
    # TODO Create a separable convolution layer using the separable_conv2d_batchnorm() function.
    output_layer = separable_conv2d_batchnorm(input_layer, filters, strides=strides)

    return output_layer

Decoder Block

The decoder block is comprised of three parts:

  • A bilinear upsampling layer using the upsample_bilinear() function. The current recommended factor for upsampling is set to 2.
  • A layer concatenation step. This step is similar to skip connections. You will concatenate the upsampled small_ip_layer and the large_ip_layer.
  • Some (one or two) additional separable convolution layers to extract some more spatial information from prior layers.
In [10]:
def decoder_block(small_ip_layer, large_ip_layer, filters):
    
    # TODO Upsample the small input layer using the bilinear_upsample() function.
    output_layer = bilinear_upsample(small_ip_layer)
    # TODO Concatenate the upsampled and large input layers using layers.concatenate
    output_layer = layers.concatenate([output_layer, large_ip_layer])
    # TODO Add some number of separable convolution layers
    output_layer = separable_conv2d_batchnorm(output_layer, filters)
    output_layer = separable_conv2d_batchnorm(output_layer, filters)  
   
    return output_layer

Model

Now that you have the encoder and decoder blocks ready, go ahead and build your FCN architecture!

There are three steps:

  • Add encoder blocks to build the encoder layers. This is similar to how you added regular convolutional layers in your CNN lab.
  • Add a 1x1 Convolution layer using the conv2d_batchnorm() function. Remember that 1x1 Convolutions require a kernel and stride of 1.
  • Add decoder blocks for the decoder layers.
In [11]:
def fcn_model(inputs, num_classes):
    
    # TODO Add Encoder Blocks. 
    conv_in = conv2d_batchnorm(input_layer=inputs, filters=16, kernel_size=1, strides=1)
    enc_1 = encoder_block(input_layer=conv_in,  filters=32,  strides=2)
    # img_w/2 x img_h/2 x 32 => img_w/4 x img_h/4 x 64
    enc_2 = encoder_block(input_layer=enc_1,   filters=64,  strides=2)
    # img_w/4 x img_h/4 x 64 => img_w/8 x img_h/8 x 128
    enc_3_1 = encoder_block(input_layer=enc_2, filters=128, strides=2)
    # img_w/8 x img_h/8 x 128 => img_w/8 x img_h/8 x 128
    enc_3_2 = conv2d_batchnorm(input_layer=enc_3_1, filters=128, kernel_size=1, strides=1)
    # img_w/8 x img_h/8 x 128 => img_w/16 x img_h/16 x 256
    enc_4 = encoder_block(input_layer=enc_3_2, filters=256, strides=2)
    
    # Remember that with each encoder layer, the depth of your model (the number of filters) increases.

    # TODO Add 1x1 Convolution layer using conv2d_batchnorm().
    conv_1x1 = conv2d_batchnorm(input_layer=enc_4,    filters=256, kernel_size=1, strides=1)
    # img_w/16 x img_h/16 x 256 => img_w/16 x img_h/16 x 128
    conv_1x1 = conv2d_batchnorm(input_layer=conv_1x1, filters=128, kernel_size=1, strides=1)    
      
    # TODO: Add the same number of Decoder Blocks as the number of Encoder Blocks
    dec_1 = decoder_block(small_ip_layer=conv_1x1, large_ip_layer=enc_3_1,  filters=128)
    # img_w/8 x img_h/8 x 128 => img_w/4 x img_h/4 x 64
    dec_2 = decoder_block(small_ip_layer=dec_1,    large_ip_layer=enc_2,  filters=64)
    # img_w/4 x img_h/4 x 64 => img_w/2 x img_h/2 x 32
    dec_3 = decoder_block(small_ip_layer=dec_2,    large_ip_layer=enc_1,  filters=32)
    # img_w/2 x img_h/2 x 32 => img_w x img_h x num_classes
    x = decoder_block(small_ip_layer=dec_3,        large_ip_layer=inputs, filters=num_classes)   
    return layers.Conv2D(num_classes, 3, activation='softmax', padding='same')(x)

Training

The following cells will use the FCN you created and define an ouput layer based on the size of the processed image and the number of classes recognized. You will define the hyperparameters to compile and train your model.

Please Note: For this project, the helper code in data_iterator.py will resize the copter images to 160x160x3 to speed up training.

In [12]:
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""

image_hw = 160
image_shape = (image_hw, image_hw, 3)
inputs = layers.Input(image_shape)
num_classes = 3

# Call fcn_model()
output_layer = fcn_model(inputs, num_classes)

Hyperparameters

Define and tune your hyperparameters.

  • batch_size: number of training samples/images that get propagated through the network in a single pass.
  • num_epochs: number of times the entire training dataset gets propagated through the network.
  • steps_per_epoch: number of batches of training images that go through the network in 1 epoch. We have provided you with a default value. One recommended value to try would be based on the total number of images in training dataset divided by the batch_size.
  • validation_steps: number of batches of validation images that go through the network in 1 epoch. This is similar to steps_per_epoch, except validation_steps is for the validation dataset. We have provided you with a default value for this as well.
  • workers: maximum number of processes to spin up. This can affect your training speed and is dependent on your hardware. We have provided a recommended value to work with.
In [13]:
learning_rate = 0.001
batch_size = 100
num_epochs = 200
steps_per_epoch = 50
validation_steps = 12
workers = 4
In [14]:
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""

from workspace_utils import active_session
# Keeping Your Session Active
with active_session():
    # Define the Keras model and compile it for training
    model = models.Model(inputs=inputs, outputs=output_layer)

    model.compile(optimizer=keras.optimizers.Adam(learning_rate), loss='categorical_crossentropy')

    # Data iterators for loading the training and validation data
    train_iter = data_iterator.BatchIteratorSimple(batch_size=batch_size,
                                                   data_folder=os.path.join('..', 'data', 'train'),
                                                   image_shape=image_shape,
                                                   shift_aug=True)

    val_iter = data_iterator.BatchIteratorSimple(batch_size=batch_size,
                                                 data_folder=os.path.join('..', 'data', 'validation'),
                                                 image_shape=image_shape)

    logger_cb = plotting_tools.LoggerPlotter()
    callbacks = [logger_cb]

    model.fit_generator(train_iter,
                        steps_per_epoch = steps_per_epoch, # the number of batches per epoch,
                        epochs = num_epochs, # the number of epochs to train for,
                        validation_data = val_iter, # validation iterator
                        validation_steps = validation_steps, # the number of batches to validate on
                        callbacks=callbacks,
                        workers = workers)
Epoch 1/200
49/50 [============================>.] - ETA: 1s - loss: 1.0163
50/50 [==============================] - 73s - loss: 1.0135 - val_loss: 0.9008
Epoch 2/200
49/50 [============================>.] - ETA: 1s - loss: 0.7138
50/50 [==============================] - 67s - loss: 0.7102 - val_loss: 0.5508
Epoch 3/200
49/50 [============================>.] - ETA: 1s - loss: 0.3804
50/50 [==============================] - 66s - loss: 0.3780 - val_loss: 0.2589
Epoch 4/200
49/50 [============================>.] - ETA: 1s - loss: 0.1930
50/50 [==============================] - 67s - loss: 0.1919 - val_loss: 0.1577
Epoch 5/200
49/50 [============================>.] - ETA: 1s - loss: 0.1161
50/50 [==============================] - 66s - loss: 0.1156 - val_loss: 0.1268
Epoch 6/200
49/50 [============================>.] - ETA: 1s - loss: 0.0830
50/50 [==============================] - 66s - loss: 0.0827 - val_loss: 0.1175
Epoch 7/200
49/50 [============================>.] - ETA: 1s - loss: 0.0667
50/50 [==============================] - 66s - loss: 0.0666 - val_loss: 0.1098
Epoch 8/200
49/50 [============================>.] - ETA: 1s - loss: 0.0576
50/50 [==============================] - 66s - loss: 0.0574 - val_loss: 0.1042
Epoch 9/200
49/50 [============================>.] - ETA: 1s - loss: 0.0427
50/50 [==============================] - 66s - loss: 0.0426 - val_loss: 0.0914
Epoch 11/200
49/50 [============================>.] - ETA: 1s - loss: 0.0376
50/50 [==============================] - 66s - loss: 0.0375 - val_loss: 0.0752
Epoch 12/200
49/50 [============================>.] - ETA: 1s - loss: 0.0346
50/50 [==============================] - 66s - loss: 0.0346 - val_loss: 0.0538
Epoch 13/200
49/50 [============================>.] - ETA: 1s - loss: 0.0315
50/50 [==============================] - 66s - loss: 0.0314 - val_loss: 0.0477
Epoch 14/200
49/50 [============================>.] - ETA: 1s - loss: 0.0284
50/50 [==============================] - 66s - loss: 0.0283 - val_loss: 0.0386
Epoch 15/200
49/50 [============================>.] - ETA: 1s - loss: 0.0283
50/50 [==============================] - 66s - loss: 0.0282 - val_loss: 0.0406
Epoch 16/200
49/50 [============================>.] - ETA: 1s - loss: 0.0264
50/50 [==============================] - 65s - loss: 0.0264 - val_loss: 0.0388
Epoch 17/200
49/50 [============================>.] - ETA: 1s - loss: 0.0257
50/50 [==============================] - 66s - loss: 0.0258 - val_loss: 0.0497
Epoch 18/200
49/50 [============================>.] - ETA: 1s - loss: 0.0272
50/50 [==============================] - 66s - loss: 0.0271 - val_loss: 0.0370
Epoch 19/200
49/50 [============================>.] - ETA: 1s - loss: 0.0255
50/50 [==============================] - 67s - loss: 0.0255 - val_loss: 0.0370
Epoch 20/200
49/50 [============================>.] - ETA: 1s - loss: 0.0235
50/50 [==============================] - 66s - loss: 0.0235 - val_loss: 0.0414
Epoch 21/200
49/50 [============================>.] - ETA: 1s - loss: 0.0227
50/50 [==============================] - 66s - loss: 0.0227 - val_loss: 0.0360
Epoch 22/200
49/50 [============================>.] - ETA: 1s - loss: 0.0243
50/50 [==============================] - 67s - loss: 0.0245 - val_loss: 0.0770
Epoch 23/200
49/50 [============================>.] - ETA: 1s - loss: 0.0266
50/50 [==============================] - 66s - loss: 0.0266 - val_loss: 0.0421
Epoch 24/200
49/50 [============================>.] - ETA: 1s - loss: 0.0232
50/50 [==============================] - 66s - loss: 0.0232 - val_loss: 0.0409
Epoch 25/200
49/50 [============================>.] - ETA: 1s - loss: 0.0216
50/50 [==============================] - 66s - loss: 0.0215 - val_loss: 0.0340
Epoch 26/200
49/50 [============================>.] - ETA: 1s - loss: 0.0205
50/50 [==============================] - 66s - loss: 0.0205 - val_loss: 0.0398
Epoch 27/200
49/50 [============================>.] - ETA: 1s - loss: 0.0315
50/50 [==============================] - 65s - loss: 0.0314 - val_loss: 0.0912
Epoch 28/200
49/50 [============================>.] - ETA: 1s - loss: 0.0251
50/50 [==============================] - 66s - loss: 0.0250 - val_loss: 0.0397
Epoch 29/200
49/50 [============================>.] - ETA: 1s - loss: 0.0217
50/50 [==============================] - 66s - loss: 0.0217 - val_loss: 0.0340
Epoch 30/200
49/50 [============================>.] - ETA: 1s - loss: 0.0210
50/50 [==============================] - 66s - loss: 0.0210 - val_loss: 0.0397
Epoch 31/200
49/50 [============================>.] - ETA: 1s - loss: 0.0199
50/50 [==============================] - 66s - loss: 0.0199 - val_loss: 0.0359
Epoch 32/200
49/50 [============================>.] - ETA: 1s - loss: 0.0202
50/50 [==============================] - 65s - loss: 0.0202 - val_loss: 0.0365
Epoch 33/200
49/50 [============================>.] - ETA: 1s - loss: 0.0194
50/50 [==============================] - 66s - loss: 0.0194 - val_loss: 0.0284
Epoch 34/200
49/50 [============================>.] - ETA: 1s - loss: 0.0186
50/50 [==============================] - 66s - loss: 0.0186 - val_loss: 0.0339
Epoch 35/200
49/50 [============================>.] - ETA: 1s - loss: 0.0190
50/50 [==============================] - 66s - loss: 0.0190 - val_loss: 0.0288
Epoch 36/200
49/50 [============================>.] - ETA: 1s - loss: 0.0175
50/50 [==============================] - 66s - loss: 0.0175 - val_loss: 0.0302
Epoch 39/200
49/50 [============================>.] - ETA: 1s - loss: 0.0174
50/50 [==============================] - 66s - loss: 0.0174 - val_loss: 0.0269
Epoch 40/200
49/50 [============================>.] - ETA: 1s - loss: 0.0183
50/50 [==============================] - 66s - loss: 0.0182 - val_loss: 0.0358
Epoch 41/200
49/50 [============================>.] - ETA: 1s - loss: 0.0175
50/50 [==============================] - 66s - loss: 0.0175 - val_loss: 0.0446
Epoch 42/200
49/50 [============================>.] - ETA: 1s - loss: 0.0172
50/50 [==============================] - 65s - loss: 0.0172 - val_loss: 0.0293
Epoch 43/200
49/50 [============================>.] - ETA: 1s - loss: 0.0168
50/50 [==============================] - 66s - loss: 0.0168 - val_loss: 0.0344
Epoch 44/200
49/50 [============================>.] - ETA: 1s - loss: 0.0164
50/50 [==============================] - 66s - loss: 0.0164 - val_loss: 0.0393
Epoch 45/200
49/50 [============================>.] - ETA: 1s - loss: 0.0163
50/50 [==============================] - 66s - loss: 0.0163 - val_loss: 0.0410
Epoch 46/200
49/50 [============================>.] - ETA: 1s - loss: 0.0161
50/50 [==============================] - 66s - loss: 0.0161 - val_loss: 0.0343
Epoch 47/200
49/50 [============================>.] - ETA: 1s - loss: 0.0160
50/50 [==============================] - 67s - loss: 0.0160 - val_loss: 0.0345
Epoch 48/200
49/50 [============================>.] - ETA: 1s - loss: 0.0177
50/50 [==============================] - 65s - loss: 0.0177 - val_loss: 0.0667
Epoch 49/200
49/50 [============================>.] - ETA: 1s - loss: 0.0246
50/50 [==============================] - 66s - loss: 0.0245 - val_loss: 0.0402
Epoch 50/200
49/50 [============================>.] - ETA: 1s - loss: 0.0198
50/50 [==============================] - 66s - loss: 0.0198 - val_loss: 0.0345
Epoch 51/200
49/50 [============================>.] - ETA: 1s - loss: 0.0177
50/50 [==============================] - 66s - loss: 0.0177 - val_loss: 0.0317
Epoch 52/200
49/50 [============================>.] - ETA: 1s - loss: 0.0170
50/50 [==============================] - 66s - loss: 0.0170 - val_loss: 0.0330
Epoch 53/200
49/50 [============================>.] - ETA: 1s - loss: 0.0160
50/50 [==============================] - 66s - loss: 0.0160 - val_loss: 0.0303
Epoch 54/200
49/50 [============================>.] - ETA: 1s - loss: 0.0171
50/50 [==============================] - 67s - loss: 0.0170 - val_loss: 0.0480
Epoch 55/200
49/50 [============================>.] - ETA: 1s - loss: 0.0197
50/50 [==============================] - 67s - loss: 0.0196 - val_loss: 0.0389
Epoch 56/200
49/50 [============================>.] - ETA: 1s - loss: 0.0180
50/50 [==============================] - 67s - loss: 0.0180 - val_loss: 0.0255
Epoch 57/200
49/50 [============================>.] - ETA: 1s - loss: 0.0158
50/50 [==============================] - 67s - loss: 0.0158 - val_loss: 0.0345
Epoch 58/200
49/50 [============================>.] - ETA: 1s - loss: 0.0159
50/50 [==============================] - 66s - loss: 0.0159 - val_loss: 0.0343
Epoch 59/200
49/50 [============================>.] - ETA: 1s - loss: 0.0170
50/50 [==============================] - 66s - loss: 0.0170 - val_loss: 0.0293
Epoch 60/200
49/50 [============================>.] - ETA: 1s - loss: 0.0149
50/50 [==============================] - 67s - loss: 0.0150 - val_loss: 0.0422
Epoch 61/200
49/50 [============================>.] - ETA: 1s - loss: 0.0147
50/50 [==============================] - 66s - loss: 0.0148 - val_loss: 0.0311
Epoch 62/200
49/50 [============================>.] - ETA: 1s - loss: 0.0140
50/50 [==============================] - 67s - loss: 0.0140 - val_loss: 0.0371
Epoch 63/200
49/50 [============================>.] - ETA: 1s - loss: 0.0140
50/50 [==============================] - 66s - loss: 0.0139 - val_loss: 0.0332
Epoch 64/200
49/50 [============================>.] - ETA: 1s - loss: 0.0137
50/50 [==============================] - 66s - loss: 0.0137 - val_loss: 0.0308
Epoch 65/200
49/50 [============================>.] - ETA: 1s - loss: 0.0135
50/50 [==============================] - 67s - loss: 0.0135 - val_loss: 0.0287
Epoch 66/200
49/50 [============================>.] - ETA: 1s - loss: 0.0133
50/50 [==============================] - 66s - loss: 0.0133 - val_loss: 0.0324
Epoch 67/200
49/50 [============================>.] - ETA: 1s - loss: 0.0131
50/50 [==============================] - 66s - loss: 0.0131 - val_loss: 0.0332
Epoch 68/200
49/50 [============================>.] - ETA: 1s - loss: 0.0127
50/50 [==============================] - 66s - loss: 0.0127 - val_loss: 0.0311
Epoch 69/200
49/50 [============================>.] - ETA: 1s - loss: 0.0126
50/50 [==============================] - 66s - loss: 0.0127 - val_loss: 0.0345
Epoch 70/200
49/50 [============================>.] - ETA: 1s - loss: 0.0125
50/50 [==============================] - 66s - loss: 0.0125 - val_loss: 0.0335
Epoch 71/200
49/50 [============================>.] - ETA: 1s - loss: 0.0123
50/50 [==============================] - 67s - loss: 0.0122 - val_loss: 0.0239
Epoch 72/200
49/50 [============================>.] - ETA: 1s - loss: 0.0121
50/50 [==============================] - 66s - loss: 0.0121 - val_loss: 0.0405
Epoch 73/200
49/50 [============================>.] - ETA: 1s - loss: 0.0122
50/50 [==============================] - 67s - loss: 0.0122 - val_loss: 0.0342
Epoch 74/200
49/50 [============================>.] - ETA: 1s - loss: 0.0120
50/50 [==============================] - 66s - loss: 0.0120 - val_loss: 0.0370
Epoch 75/200
49/50 [============================>.] - ETA: 1s - loss: 0.0118
50/50 [==============================] - 67s - loss: 0.0118 - val_loss: 0.0229
Epoch 76/200
49/50 [============================>.] - ETA: 1s - loss: 0.0117
50/50 [==============================] - 67s - loss: 0.0117 - val_loss: 0.0331
Epoch 77/200
49/50 [============================>.] - ETA: 1s - loss: 0.0176
50/50 [==============================] - 66s - loss: 0.0176 - val_loss: 0.0363
Epoch 80/200
49/50 [============================>.] - ETA: 1s - loss: 0.0152
50/50 [==============================] - 66s - loss: 0.0151 - val_loss: 0.0288
Epoch 81/200
49/50 [============================>.] - ETA: 1s - loss: 0.0138
50/50 [==============================] - 66s - loss: 0.0138 - val_loss: 0.0201
Epoch 82/200
49/50 [============================>.] - ETA: 1s - loss: 0.0131
50/50 [==============================] - 67s - loss: 0.0131 - val_loss: 0.0304
Epoch 83/200
49/50 [============================>.] - ETA: 1s - loss: 0.0129
50/50 [==============================] - 67s - loss: 0.0129 - val_loss: 0.0280
Epoch 84/200
49/50 [============================>.] - ETA: 1s - loss: 0.0123
50/50 [==============================] - 65s - loss: 0.0123 - val_loss: 0.0321
Epoch 85/200
49/50 [============================>.] - ETA: 1s - loss: 0.0122
50/50 [==============================] - 66s - loss: 0.0122 - val_loss: 0.0312
Epoch 86/200
49/50 [============================>.] - ETA: 1s - loss: 0.0118
50/50 [==============================] - 66s - loss: 0.0118 - val_loss: 0.0399
Epoch 87/200
49/50 [============================>.] - ETA: 1s - loss: 0.0117
50/50 [==============================] - 67s - loss: 0.0117 - val_loss: 0.0346
Epoch 88/200
49/50 [============================>.] - ETA: 1s - loss: 0.0117
50/50 [==============================] - 67s - loss: 0.0117 - val_loss: 0.0321
Epoch 89/200
49/50 [============================>.] - ETA: 1s - loss: 0.0115
50/50 [==============================] - 66s - loss: 0.0115 - val_loss: 0.0201
Epoch 90/200
49/50 [============================>.] - ETA: 1s - loss: 0.0114
50/50 [==============================] - 66s - loss: 0.0114 - val_loss: 0.0398
Epoch 91/200
49/50 [============================>.] - ETA: 1s - loss: 0.0115
50/50 [==============================] - 67s - loss: 0.0115 - val_loss: 0.0317
Epoch 92/200
49/50 [============================>.] - ETA: 1s - loss: 0.0111
50/50 [==============================] - 66s - loss: 0.0112 - val_loss: 0.0311
Epoch 93/200
49/50 [============================>.] - ETA: 1s - loss: 0.0112
50/50 [==============================] - 66s - loss: 0.0112 - val_loss: 0.0310
Epoch 94/200
49/50 [============================>.] - ETA: 1s - loss: 0.0114
50/50 [==============================] - 67s - loss: 0.0114 - val_loss: 0.0334
Epoch 95/200
49/50 [============================>.] - ETA: 1s - loss: 0.0111
50/50 [==============================] - 66s - loss: 0.0112 - val_loss: 0.0419
Epoch 96/200
49/50 [============================>.] - ETA: 1s - loss: 0.0110
50/50 [==============================] - 66s - loss: 0.0110 - val_loss: 0.0353
Epoch 97/200
49/50 [============================>.] - ETA: 1s - loss: 0.0159
50/50 [==============================] - 67s - loss: 0.0159 - val_loss: 0.0702
Epoch 98/200
49/50 [============================>.] - ETA: 1s - loss: 0.0177
50/50 [==============================] - 66s - loss: 0.0177 - val_loss: 0.0323
Epoch 99/200
49/50 [============================>.] - ETA: 1s - loss: 0.0137
50/50 [==============================] - 66s - loss: 0.0137 - val_loss: 0.0250
Epoch 100/200
49/50 [============================>.] - ETA: 1s - loss: 0.0115
50/50 [==============================] - 66s - loss: 0.0115 - val_loss: 0.0333
Epoch 103/200
49/50 [============================>.] - ETA: 1s - loss: 0.0113
50/50 [==============================] - 66s - loss: 0.0113 - val_loss: 0.0302
Epoch 104/200
49/50 [============================>.] - ETA: 1s - loss: 0.0111
50/50 [==============================] - 66s - loss: 0.0110 - val_loss: 0.0268
Epoch 105/200
49/50 [============================>.] - ETA: 1s - loss: 0.0109
50/50 [==============================] - 65s - loss: 0.0109 - val_loss: 0.0358
Epoch 106/200
49/50 [============================>.] - ETA: 1s - loss: 0.0110
50/50 [==============================] - 66s - loss: 0.0110 - val_loss: 0.0314
Epoch 107/200
49/50 [============================>.] - ETA: 1s - loss: 0.0107
50/50 [==============================] - 66s - loss: 0.0107 - val_loss: 0.0336
Epoch 108/200
49/50 [============================>.] - ETA: 1s - loss: 0.0125
50/50 [==============================] - 67s - loss: 0.0125 - val_loss: 0.0469
Epoch 109/200
49/50 [============================>.] - ETA: 1s - loss: 0.0139
50/50 [==============================] - 66s - loss: 0.0139 - val_loss: 0.0310
Epoch 110/200
49/50 [============================>.] - ETA: 1s - loss: 0.0120
50/50 [==============================] - 66s - loss: 0.0120 - val_loss: 0.0228
Epoch 111/200
49/50 [============================>.] - ETA: 1s - loss: 0.0115
50/50 [==============================] - 65s - loss: 0.0115 - val_loss: 0.0273
Epoch 112/200
49/50 [============================>.] - ETA: 1s - loss: 0.0110
50/50 [==============================] - 67s - loss: 0.0110 - val_loss: 0.0309
Epoch 113/200
49/50 [============================>.] - ETA: 1s - loss: 0.0108
50/50 [==============================] - 66s - loss: 0.0108 - val_loss: 0.0363
Epoch 114/200
49/50 [============================>.] - ETA: 1s - loss: 0.0106
50/50 [==============================] - 67s - loss: 0.0106 - val_loss: 0.0264
Epoch 115/200
49/50 [============================>.] - ETA: 1s - loss: 0.0106
50/50 [==============================] - 67s - loss: 0.0106 - val_loss: 0.0308
Epoch 116/200
49/50 [============================>.] - ETA: 1s - loss: 0.0105
50/50 [==============================] - 66s - loss: 0.0105 - val_loss: 0.0325
Epoch 117/200
49/50 [============================>.] - ETA: 1s - loss: 0.0104
50/50 [==============================] - 67s - loss: 0.0104 - val_loss: 0.0333
Epoch 118/200
49/50 [============================>.] - ETA: 1s - loss: 0.0101
50/50 [==============================] - 67s - loss: 0.0101 - val_loss: 0.0319
Epoch 119/200
49/50 [============================>.] - ETA: 1s - loss: 0.0111
50/50 [==============================] - 67s - loss: 0.0111 - val_loss: 0.0340
Epoch 120/200
49/50 [============================>.] - ETA: 1s - loss: 0.0108
50/50 [==============================] - 66s - loss: 0.0108 - val_loss: 0.0264
Epoch 121/200
49/50 [============================>.] - ETA: 1s - loss: 0.0102
50/50 [==============================] - 65s - loss: 0.0102 - val_loss: 0.0317
Epoch 122/200
49/50 [============================>.] - ETA: 1s - loss: 0.0101
50/50 [==============================] - 66s - loss: 0.0102 - val_loss: 0.0226
Epoch 123/200
49/50 [============================>.] - ETA: 1s - loss: 0.0101
50/50 [==============================] - 66s - loss: 0.0101 - val_loss: 0.0447
Epoch 124/200
49/50 [============================>.] - ETA: 1s - loss: 0.0100
50/50 [==============================] - 66s - loss: 0.0100 - val_loss: 0.0327
Epoch 125/200
49/50 [============================>.] - ETA: 1s - loss: 0.0099
50/50 [==============================] - 66s - loss: 0.0099 - val_loss: 0.0263
Epoch 126/200
49/50 [============================>.] - ETA: 1s - loss: 0.0099
50/50 [==============================] - 66s - loss: 0.0098 - val_loss: 0.0342
Epoch 127/200
49/50 [============================>.] - ETA: 1s - loss: 0.0098
50/50 [==============================] - 66s - loss: 0.0098 - val_loss: 0.0338
Epoch 128/200
49/50 [============================>.] - ETA: 1s - loss: 0.0098
50/50 [==============================] - 67s - loss: 0.0098 - val_loss: 0.0293
Epoch 129/200
49/50 [============================>.] - ETA: 1s - loss: 0.0097
50/50 [==============================] - 67s - loss: 0.0097 - val_loss: 0.0378
Epoch 130/200
49/50 [============================>.] - ETA: 1s - loss: 0.0098
50/50 [==============================] - 67s - loss: 0.0098 - val_loss: 0.0335
Epoch 131/200
49/50 [============================>.] - ETA: 1s - loss: 0.0095
50/50 [==============================] - 66s - loss: 0.0095 - val_loss: 0.0356
Epoch 132/200
49/50 [============================>.] - ETA: 1s - loss: 0.0096
50/50 [==============================] - 67s - loss: 0.0096 - val_loss: 0.0311
Epoch 133/200
49/50 [============================>.] - ETA: 1s - loss: 0.0097
50/50 [==============================] - 67s - loss: 0.0097 - val_loss: 0.0332
Epoch 134/200
49/50 [============================>.] - ETA: 1s - loss: 0.0097
50/50 [==============================] - 66s - loss: 0.0097 - val_loss: 0.0321
Epoch 135/200
49/50 [============================>.] - ETA: 1s - loss: 0.0098
50/50 [==============================] - 67s - loss: 0.0098 - val_loss: 0.0381
Epoch 136/200
49/50 [============================>.] - ETA: 1s - loss: 0.0096
50/50 [==============================] - 67s - loss: 0.0096 - val_loss: 0.0336
Epoch 137/200
49/50 [============================>.] - ETA: 1s - loss: 0.0094
50/50 [==============================] - 66s - loss: 0.0094 - val_loss: 0.0249
Epoch 138/200
49/50 [============================>.] - ETA: 1s - loss: 0.0093
50/50 [==============================] - 66s - loss: 0.0094 - val_loss: 0.0203
Epoch 139/200
49/50 [============================>.] - ETA: 1s - loss: 0.0093
50/50 [==============================] - 67s - loss: 0.0093 - val_loss: 0.0345
Epoch 140/200
49/50 [============================>.] - ETA: 1s - loss: 0.0093
50/50 [==============================] - 66s - loss: 0.0093 - val_loss: 0.0411
Epoch 141/200
49/50 [============================>.] - ETA: 1s - loss: 0.0094
50/50 [==============================] - 66s - loss: 0.0094 - val_loss: 0.0354
Epoch 142/200
49/50 [============================>.] - ETA: 1s - loss: 0.0093
50/50 [==============================] - 65s - loss: 0.0092 - val_loss: 0.0352
Epoch 143/200
49/50 [============================>.] - ETA: 1s - loss: 0.0093
50/50 [==============================] - 66s - loss: 0.0093 - val_loss: 0.0247
Epoch 144/200
49/50 [============================>.] - ETA: 1s - loss: 0.0093
50/50 [==============================] - 66s - loss: 0.0093 - val_loss: 0.0338
Epoch 145/200
49/50 [============================>.] - ETA: 1s - loss: 0.0093
50/50 [==============================] - 67s - loss: 0.0092 - val_loss: 0.0366
Epoch 146/200
49/50 [============================>.] - ETA: 1s - loss: 0.0093
50/50 [==============================] - 67s - loss: 0.0093 - val_loss: 0.0406
Epoch 147/200
49/50 [============================>.] - ETA: 1s - loss: 0.0091
50/50 [==============================] - 66s - loss: 0.0091 - val_loss: 0.0465
Epoch 148/200
49/50 [============================>.] - ETA: 1s - loss: 0.0090
50/50 [==============================] - 66s - loss: 0.0090 - val_loss: 0.0350
Epoch 149/200
49/50 [============================>.] - ETA: 1s - loss: 0.0091
50/50 [==============================] - 67s - loss: 0.0091 - val_loss: 0.0252
Epoch 150/200
49/50 [============================>.] - ETA: 1s - loss: 0.0091
50/50 [==============================] - 67s - loss: 0.0091 - val_loss: 0.0388
Epoch 151/200
49/50 [============================>.] - ETA: 1s - loss: 0.0092
50/50 [==============================] - 67s - loss: 0.0092 - val_loss: 0.0349
Epoch 152/200
49/50 [============================>.] - ETA: 1s - loss: 0.0091
50/50 [==============================] - 66s - loss: 0.0091 - val_loss: 0.0352
Epoch 153/200
49/50 [============================>.] - ETA: 1s - loss: 0.0090
50/50 [==============================] - 67s - loss: 0.0090 - val_loss: 0.0413
Epoch 154/200
49/50 [============================>.] - ETA: 1s - loss: 0.0091
50/50 [==============================] - 66s - loss: 0.0091 - val_loss: 0.0363
Epoch 155/200
49/50 [============================>.] - ETA: 1s - loss: 0.0089
50/50 [==============================] - 66s - loss: 0.0089 - val_loss: 0.0361
Epoch 156/200
49/50 [============================>.] - ETA: 1s - loss: 0.0090
50/50 [==============================] - 66s - loss: 0.0090 - val_loss: 0.0422
Epoch 157/200
49/50 [============================>.] - ETA: 1s - loss: 0.0088
50/50 [==============================] - 67s - loss: 0.0088 - val_loss: 0.0361
Epoch 158/200
49/50 [============================>.] - ETA: 1s - loss: 0.0090
50/50 [==============================] - 66s - loss: 0.0090 - val_loss: 0.0448
Epoch 159/200
49/50 [============================>.] - ETA: 1s - loss: 0.0089
50/50 [==============================] - 66s - loss: 0.0089 - val_loss: 0.0345
Epoch 160/200
49/50 [============================>.] - ETA: 1s - loss: 0.0090
50/50 [==============================] - 66s - loss: 0.0090 - val_loss: 0.0362
Epoch 161/200
49/50 [============================>.] - ETA: 1s - loss: 0.0089
50/50 [==============================] - 66s - loss: 0.0088 - val_loss: 0.0389
Epoch 162/200
49/50 [============================>.] - ETA: 1s - loss: 0.0089
50/50 [==============================] - 67s - loss: 0.0089 - val_loss: 0.0489
Epoch 163/200
49/50 [============================>.] - ETA: 1s - loss: 0.0088
50/50 [==============================] - 66s - loss: 0.0088 - val_loss: 0.0368
Epoch 164/200
49/50 [============================>.] - ETA: 1s - loss: 0.0089
50/50 [==============================] - 67s - loss: 0.0089 - val_loss: 0.0319
Epoch 165/200
49/50 [============================>.] - ETA: 1s - loss: 0.0087
50/50 [==============================] - 66s - loss: 0.0087 - val_loss: 0.0298
Epoch 166/200
49/50 [============================>.] - ETA: 1s - loss: 0.0089
50/50 [==============================] - 66s - loss: 0.0089 - val_loss: 0.0378
Epoch 167/200
49/50 [============================>.] - ETA: 1s - loss: 0.0117
50/50 [==============================] - 66s - loss: 0.0118 - val_loss: 0.0533
Epoch 168/200
49/50 [============================>.] - ETA: 1s - loss: 0.0197
50/50 [==============================] - 66s - loss: 0.0197 - val_loss: 0.0657
Epoch 169/200
49/50 [============================>.] - ETA: 1s - loss: 0.0154
50/50 [==============================] - 67s - loss: 0.0154 - val_loss: 0.0404
Epoch 170/200
49/50 [============================>.] - ETA: 1s - loss: 0.0148
50/50 [==============================] - 66s - loss: 0.0147 - val_loss: 0.0316
Epoch 173/200
49/50 [============================>.] - ETA: 1s - loss: 0.0144
50/50 [==============================] - 66s - loss: 0.0144 - val_loss: 0.0318
Epoch 174/200
49/50 [============================>.] - ETA: 1s - loss: 0.0130
50/50 [==============================] - 66s - loss: 0.0130 - val_loss: 0.0219
Epoch 175/200
49/50 [============================>.] - ETA: 1s - loss: 0.0122
50/50 [==============================] - 66s - loss: 0.0122 - val_loss: 0.0304
Epoch 176/200
49/50 [============================>.] - ETA: 1s - loss: 0.0115
50/50 [==============================] - 66s - loss: 0.0115 - val_loss: 0.0198
Epoch 177/200
49/50 [============================>.] - ETA: 1s - loss: 0.0115
50/50 [==============================] - 66s - loss: 0.0114 - val_loss: 0.0323
Epoch 178/200
49/50 [============================>.] - ETA: 1s - loss: 0.0113
50/50 [==============================] - 66s - loss: 0.0113 - val_loss: 0.0312
Epoch 179/200
49/50 [============================>.] - ETA: 1s - loss: 0.0110
50/50 [==============================] - 65s - loss: 0.0109 - val_loss: 0.0308
Epoch 180/200
49/50 [============================>.] - ETA: 1s - loss: 0.0108
50/50 [==============================] - 66s - loss: 0.0108 - val_loss: 0.0390
Epoch 181/200
49/50 [============================>.] - ETA: 1s - loss: 0.0109
50/50 [==============================] - 66s - loss: 0.0109 - val_loss: 0.0311
Epoch 182/200
49/50 [============================>.] - ETA: 1s - loss: 0.0107
50/50 [==============================] - 66s - loss: 0.0108 - val_loss: 0.0426
Epoch 183/200
49/50 [============================>.] - ETA: 1s - loss: 0.0105
50/50 [==============================] - 66s - loss: 0.0105 - val_loss: 0.0335
Epoch 184/200
49/50 [============================>.] - ETA: 1s - loss: 0.0103
50/50 [==============================] - 66s - loss: 0.0103 - val_loss: 0.0320
Epoch 185/200
49/50 [============================>.] - ETA: 1s - loss: 0.0102
50/50 [==============================] - 66s - loss: 0.0102 - val_loss: 0.0295
Epoch 186/200
49/50 [============================>.] - ETA: 1s - loss: 0.0100
50/50 [==============================] - 66s - loss: 0.0100 - val_loss: 0.0381
Epoch 187/200
49/50 [============================>.] - ETA: 1s - loss: 0.0102
50/50 [==============================] - 67s - loss: 0.0102 - val_loss: 0.0317
Epoch 188/200
49/50 [============================>.] - ETA: 1s - loss: 0.0102
50/50 [==============================] - 66s - loss: 0.0102 - val_loss: 0.0300
Epoch 189/200
49/50 [============================>.] - ETA: 1s - loss: 0.0104
50/50 [==============================] - 66s - loss: 0.0104 - val_loss: 0.0398
Epoch 190/200
49/50 [============================>.] - ETA: 1s - loss: 0.0099
50/50 [==============================] - 66s - loss: 0.0099 - val_loss: 0.0324
Epoch 191/200
49/50 [============================>.] - ETA: 1s - loss: 0.0098
50/50 [==============================] - 66s - loss: 0.0099 - val_loss: 0.0300
Epoch 192/200
49/50 [============================>.] - ETA: 1s - loss: 0.0098
50/50 [==============================] - 66s - loss: 0.0098 - val_loss: 0.0445
Epoch 193/200
49/50 [============================>.] - ETA: 1s - loss: 0.0097
50/50 [==============================] - 66s - loss: 0.0097 - val_loss: 0.0328
Epoch 194/200
49/50 [============================>.] - ETA: 1s - loss: 0.0099
50/50 [==============================] - 66s - loss: 0.0100 - val_loss: 0.0454
Epoch 195/200
49/50 [============================>.] - ETA: 1s - loss: 0.0096
50/50 [==============================] - 66s - loss: 0.0096 - val_loss: 0.0323
Epoch 196/200
49/50 [============================>.] - ETA: 1s - loss: 0.0095
50/50 [==============================] - 67s - loss: 0.0095 - val_loss: 0.0337
Epoch 197/200
49/50 [============================>.] - ETA: 1s - loss: 0.0095
50/50 [==============================] - 66s - loss: 0.0095 - val_loss: 0.0172
Epoch 198/200
49/50 [============================>.] - ETA: 1s - loss: 0.0094
50/50 [==============================] - 66s - loss: 0.0094 - val_loss: 0.0483
Epoch 199/200
49/50 [============================>.] - ETA: 1s - loss: 0.0094
50/50 [==============================] - 66s - loss: 0.0093 - val_loss: 0.0342
Epoch 200/200
49/50 [============================>.] - ETA: 1s - loss: 0.0095
50/50 [==============================] - 65s - loss: 0.0095 - val_loss: 0.0403
In [15]:
# Save your trained model weights
weight_file_name = 'model_weights'
model_tools.save_network(model, weight_file_name)

Prediction

Now that you have your model trained and saved, you can make predictions on your validation dataset. These predictions can be compared to the mask images, which are the ground truth labels, to evaluate how well your model is doing under different conditions.

There are three different predictions available from the helper code provided:

  • patrol_with_targ: Test how well the network can detect the hero from a distance.
  • patrol_non_targ: Test how often the network makes a mistake and identifies the wrong person as the target.
  • following_images: Test how well the network can identify the target while following them.
In [16]:
# If you need to load a model which you previously trained you can uncomment the codeline that calls the function below.

# weight_file_name = 'model_weights'
# restored_model = model_tools.load_network(weight_file_name)

The following cell will write predictions to files and return paths to the appropriate directories. The run_num parameter is used to define or group all the data for a particular model run. You can change it for different runs. For example, 'run_1', 'run_2' etc.

In [17]:
run_num = 'run_1'

val_with_targ, pred_with_targ = model_tools.write_predictions_grade_set(model,
                                        run_num,'patrol_with_targ', 'sample_evaluation_data') 

val_no_targ, pred_no_targ = model_tools.write_predictions_grade_set(model, 
                                        run_num,'patrol_non_targ', 'sample_evaluation_data') 

val_following, pred_following = model_tools.write_predictions_grade_set(model,
                                        run_num,'following_images', 'sample_evaluation_data')

Now lets look at your predictions, and compare them to the ground truth labels and original images. Run each of the following cells to visualize some sample images from the predictions in the validation set.

In [18]:
# images while following the target
im_files = plotting_tools.get_im_file_sample('sample_evaluation_data','following_images', run_num) 
for i in range(3):
    im_tuple = plotting_tools.load_images(im_files[i])
    plotting_tools.show_images(im_tuple)
    
In [19]:
# images while at patrol without target
im_files = plotting_tools.get_im_file_sample('sample_evaluation_data','patrol_non_targ', run_num) 
for i in range(3):
    im_tuple = plotting_tools.load_images(im_files[i])
    plotting_tools.show_images(im_tuple)
 
In [20]:
   
# images while at patrol with target
im_files = plotting_tools.get_im_file_sample('sample_evaluation_data','patrol_with_targ', run_num) 
for i in range(3):
    im_tuple = plotting_tools.load_images(im_files[i])
    plotting_tools.show_images(im_tuple)

Evaluation

Evaluate your model! The following cells include several different scores to help you evaluate your model under the different conditions discussed during the Prediction step.

In [21]:
# Scores for while the quad is following behind the target. 
true_pos1, false_pos1, false_neg1, iou1 = scoring_utils.score_run_iou(val_following, pred_following)
number of validation samples intersection over the union evaulated on 542
average intersection over union for background is 0.9959395360040365
average intersection over union for other people is 0.3820159722936076
average intersection over union for the hero is 0.9211558050954375
number true positives: 539, number false positives: 0, number false negatives: 0
In [22]:
# Scores for images while the quad is on patrol and the target is not visable
true_pos2, false_pos2, false_neg2, iou2 = scoring_utils.score_run_iou(val_no_targ, pred_no_targ)
number of validation samples intersection over the union evaulated on 270
average intersection over union for background is 0.9880272313720647
average intersection over union for other people is 0.7641420676715759
average intersection over union for the hero is 0.0
number true positives: 0, number false positives: 36, number false negatives: 0
In [23]:
# This score measures how well the neural network can detect the target from far away
true_pos3, false_pos3, false_neg3, iou3 = scoring_utils.score_run_iou(val_with_targ, pred_with_targ)
number of validation samples intersection over the union evaulated on 322
average intersection over union for background is 0.9964543518161035
average intersection over union for other people is 0.4622521142308116
average intersection over union for the hero is 0.22730754678862605
number true positives: 124, number false positives: 0, number false negatives: 177
In [24]:
# Sum all the true positives, etc from the three datasets to get a weight for the score
true_pos = true_pos1 + true_pos2 + true_pos3
false_pos = false_pos1 + false_pos2 + false_pos3
false_neg = false_neg1 + false_neg2 + false_neg3

weight = true_pos/(true_pos+false_neg+false_pos)
print(weight)
0.7568493150684932
In [25]:
# The IoU for the dataset that never includes the hero is excluded from grading
final_IoU = (iou1 + iou3)/2
print(final_IoU)
0.574231675942
In [26]:
# And the final grade score is 
final_score = final_IoU * weight
print(final_score)
0.434606850627
In [ ]: